Method and Device for Removing Objects from a Video Feed in Real-time

ABSTRACT

According to embodiments of the invention, systems, methods and devices are directed to removal of objects in a video feed that are captured continuously through video sources in real-time. Various embodiments of the invention implement several novel features, including, in one embodiment, a device that is coupled to cameras has an input device for selecting the objects to be removed from the feed. The selected object is removed and replaced with pixels neighboring the location where the pixels were taken from. Contour may also be controlled via the input, and sensitivity may be adjusted based on the application. Removed objects may also be selectively reintroduced into the video feed via the input device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 14/225,381,filed Mar. 25, 2014.

FIELD OF THE INVENTION

This invention generally relates to multimedia. Specifically, thisinvention relates to removing unwanted and/or sensitiveobjects/images/people from a video feed in real-time.

BACKGROUND OF THE INVENTION

Video capturing and surveillance has segmented itself into many facetsof everyday life. No matter where an individual goes or what anindividual does, there is always a video being captured or generated.The most common reason for the multitude of cameras in public spaces issecurity. However, the growing number of cameras has led to an increasein privacy concerns.

Video editing is well known in the art. Video processing techniques arecapable of processing of images so that some images look verydifferently than the original source. There is no question that videoprocessing can be used in protecting certain information from beingdisclosed to the general public or to the wrong audience. For example,these techniques are used extensively in security checking in virtuallyevery building where security is a concerned, such as hospitals, policestations, and airports. In most instances, some of these videos arecaptured continuously, 24 hours a day. The challenge is, there arehardly any technologies to remove certain parts of a video due toprivacy concerns. For example, in a video captured in an airport,certain individuals may appear on camera who posed a security risk ormay be at risk if their whereabouts are known to others. To protectthose people in terms of privacy, there exists a need in the art toremove certain moving objects from a live video feed.

SUMMARY OF THE INVENTION

According to embodiments of the invention, systems, methods and devicesare directed to removal of objects in a video feed that are capturedcontinuously through video sources in real-time. Various embodiments ofthe invention implement several novel features, including, in oneembodiment, a device that is coupled to cameras has an input device forselecting the objects to be removed from the feed. The selected objectis removed and replaced with pixels neighboring the location where thepixels were taken from. Contour may also be controlled via the input,and sensitivity may be adjusted based on the application. Removedobjects may also be selectively reintroduced into the video feed via theinput device.

In an embodiment of the disclosed invention, a multimedia black box isused for protecting certain sensitive information from display byremoving selective objects from video streams captured by cameras inreal-time. The multimedia black box may have one or more of thefollowing components: a) a first input element connected to a firstdevice that has a display, a camera, a video and audio input interfaceand a video and audio output interface, the first device being atraditional image or video capturing device; b) a second input elementconnected to a second device that has a display, a camera, a video andaudio input interface and a video and audio output interface, the seconddevice being a traditional image or video capturing device; c) a firstoutput element connected to the first device for outputting processedsignals back to the first device; d) a second output element connectedto the second device for outputting processed signals back to the seconddevice; and e) a touch screen having a user interface operable to selectobjects on a video stream being captured by the first and seconddevices:

The user interface facilitates and/or carries out the following: a)video signals captured from both the first device and the second deviceare transmitted simultaneously to the black box via the first and secondinput elements; b) the video signals can be paused when the signalsarrive at the touch screen of the black box such that contours ofobjects are definable by a user to be removed; c) the defined contour ofobjects and the video are further sent securely to a remote server via anetwork; d) the video signals of both the first and second devices arescreened in real-time to detect and remove any instances of the definedcontour of objects are removed from each of the video signals andmissing areas vacated by the removed objects are filled usingneighboring pixels; and e) the processed video signals are subsequentlysent back to the respective devices by the black box through the firstand second outputs, the processed videos signals appearing on thedisplays of the first and second devices.

In a further embodiment, sensitivity of the defined contour of objectsmay be configurable via a remote server, so that one object can bescreened from the video signals when the detected object reaches a lowermatching sensitivity level whereas another object stays in the videosignals when the object fails to reach a defined matching sensitivitylevel. Further, other parameters of the black box may be configurablevia a web page, the parameters including choices of processing means,locations of the remote server or the cloud, authentication information,security certificates, and various types of processing choices of theaudio and video signals, further wherein a processing choice includesmixing the video and audio signals so that the processed video and audiosignals appear to originate from a single source.

Still further, the processed video signals may be outputted to a pairingoutput, the pairing output being an output channel having video andaudio signals that are synchronized simultaneously with a second device.The output element may be connected to both the first or second devicesthrough a KVM switch.

In still a further embodiment, the user may undo the removal of onedefined contour of object so that the one defined contour of objectreappears but the rest of the defined contour of objects remain removedfrom the video signals. Further, a user may configure parameters of theblack boxes through a web page by selecting black boxes. The parametersmay include choices of processing means, locations of the remote serveror the cloud, authentication information, security certificates, andvarious types of processing choices of the audio and video signals. Theprocessing choice may include mixing the video and audio signals so thatthe processed video and audio signals appear to originate from only asingle source. Furthermore, the first and second black boxes may belocated in separate premises. The processed video and audio signals maybe outputted to a pairing output, the pairing output being an outputchannel whose video and audio signals are synchronized simultaneouslywith a third device.

In accordance with these and other objects which will become apparenthereinafter, the invention will now be described with particularreference to the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a top plan view of an exemplary configuration according to anembodiment of the disclosed technology.

FIG. 2 shows the configuration of FIG. 1 with the selected objectremoved.

FIG. 3 is a top plan view of another exemplary configuration accordingto an embodiment of the disclosed technology.

FIG. 4 shows the configuration of FIG. 3 with the selected objectremoved.

FIG. 5 is a high-level block diagram of a microprocessor device that maybe used to carry out the disclosed technology.

DETAILED DESCRIPTION

According to embodiments of the invention, systems, methods and devicesare directed to removal of objects in a video feed that are capturedcontinuously through video sources in real-time. Various embodiments ofthe invention implement several novel features, including, in oneembodiment, a device that is coupled to cameras has an input device forselecting the objects to be removed from the feed. The selected objectis removed and replaced with pixels neighboring the location where thepixels were taken from. Contour may also be controlled via the input,and sensitivity may be adjusted based on the application. Removedobjects may also be selectively reintroduced into the video feed via theinput device.

Referring now to the figures, systems, methods and devices are providedfor selectively removing objects from a real-time video feed using ablack box device to pre-specify the contours of the object to be removedwhen detected in the feed. A “black box” may be any device having one ormore video inputs and outputs, and is used to exemplary purposes. Thus,the term “black box” is not meant to be limiting in any manner. FIG. 1is a top plan view of an exemplary configuration accord. A multimediablack box is provided for protecting certain sensitive information fromdisplay by removing selective objects 11 from video streams captured bycameras 1 and 2 in real-time. Such a device comprises a first inputelement connected to a first device that has a display, a camera 1, avideo and audio input interface and a video and audio output interface,the first device being a traditional image or video capturing device.Further, a second input element is demonstrated. It is connected to asecond device that has a display, a camera 2, a video and audio inputinterface and a video and audio output interface. As one understood byskilled in the art, the second device can be a traditional image orvideo capturing device.

Output elements are included in the black box. For example, a firstoutput element can be connected to the first device 10 for outputtingprocessed signals back to the first device, whereas a second outputelement is connected to the second device for outputting processedsignals back to the second device. FIG. 3 is a top plan view of anotherexemplary configuration according to an embodiment of the disclosedtechnology. The exemplary black box device can further include a touchscreen having a user interface operable to select objects on a videostream being captured by the first and second devices 10, wherein videosignals are captured from both the first device and the second deviceare transmitted simultaneously to the black box via the first and secondinput elements. Functions can be performed on the video signals. Forexample, the video signals can be configured to be paused when thesignals arrive at the touch screen of the black box such that contoursof objects are definable by a user to be removed. The defined contour ofobjects and the video are further sent securely to a remote server via anetwork, so that the video signals of both the first and second devices10 are screened in real-time to detect and remove any instances of thedefined contour of objects 11 are removed from each of the video signalsand missing areas vacated by the removed objects 11 are filled usingneighboring pixels, as shown on FIG. 2, which shows the configuration ofFIG. 1 with the selected object removed.

The processed video signals are subsequently sent back to the respectivedevices by the black box through the above-stated first and secondoutputs. These videos signals can appear on multiple displays of thefirst and second devices. FIG. 4 shows the configuration of FIG. 3 withthe selected object removed. According to another embodiment consistentwith the present invention, the sensitivity of the defined contour ofobjects is configurable via a remote server, so that one object can bescreened from the video signals when the detected object 14 reaches alower matching sensitivity level whereas another object 15 stays in thevideo signals when the object fails to reach a defined matchingsensitivity level.

Further into the embodiment, other parameters of the black box areconfigurable via many user interfaces such as a web page. In oneadditional embodiment, such parameters may include choices of processingmeans, locations of the remote server or the cloud, authenticationinformation, security certificates, and various types of processingchoices of the audio and video signals, further wherein a processingchoice includes mixing the video and audio signals so that the processedvideo and audio signals appear to originate from a single source.Subsequently, the processed video signals are outputted to a pairingoutput that can serve as an output channel having video and audiosignals that are synchronized simultaneously with a second device. Theembodiment can be implemented so that the output element can beconnected to both the first or second devices through a KVM switchallowing some flexibility to the black box user.

FIG. 5 is a high-level block diagram of a microprocessor device that maybe used to carry out the disclosed technology. The device 300 may or maynot be a computing device. The device 300 comprises a processor 350 thatcontrols the overall operation of a computer by executing the reader'sprogram instructions which define such operation. The device's programinstructions may be stored in a storage device 320 (e.g., magnetic disk,database) and loaded into memory 330 when execution of the console'sprogram instructions is desired. Thus, the device's operation will bedefined by its program instructions stored in memory 330 and/or storage320, and the console will be controlled by the processor 350 executingthe console's program instructions.

The device 300 may also include one or a plurality of input networkinterfaces for communicating with other devices via a network (e.g., theinternet). The device 300 further includes an electrical input interfacefor receiving power and data from a power or RFID source. The device 300may also include one or more output network interfaces 310 forcommunicating with other devices. The device 300 may also includeinput/output 340 representing devices which allow for user interactionwith a computer (e.g., display, keyboard, mouse, speakers, buttons,etc.).

One skilled in the art will recognize that an implementation of anactual device will contain other components as well, and that FIG. 5 isa high level representation of some of the components of such a devicefor illustrative purposes. It should also be understood by one skilledin the art that the devices depicted and described with respect to FIGS.1 through 4 may be implemented on a device such as is shown in FIG. 5.

While the disclosed invention has been taught with specific reference tothe above embodiments, a person having ordinary skill in the art willrecognize that changes can be made in form and detail without departingfrom the spirit and the scope of the invention. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. All changes that come within the meaning and rangeof equivalency of the claims are to be embraced within their scope.Combinations of any of the methods, systems, and devices describedherein are also contemplated and within the scope of the invention.

What is claimed is:
 1. A method for selectively removing specificobjects in real-time from a video feed, the method comprising: capturinga scene using a first and a second camera, the first and the secondcameras being placed substantially close to one another; receiving, at amultimedia device, a real-time video feed from each camera via an audioand video interface, wherein the multimedia black box has a display, aprocessor, memory and an input device; displaying, on the display, thereal-time video feeds from each of the cameras; defining and identifyingobjects present in the video feeds by detecting the contours of eachobject in three dimensions using the different perspectives of eachcamera; receiving via the input device, one or more selections ofdefined objects to be removed from an outputted video feed; and removingthe selections of defined objects from the video feed and outputting afinal real-time video feed from the multimedia black box.
 2. The methodof claim 1, wherein the sensitivity of the contour detecting may beadjusted via the input device.
 3. The method of claim 1, wherein thesensitivity of the contour detecting may be adjusted via a remoteserver.
 4. The method of claim 3, wherein the remote server isaccessible by a web page.
 5. The method of claim 1, wherein the inputdevice is a touch screen.
 6. The method of claim 1, wherein the finalreal-time video feed is outputted via a KVM switch.
 7. The method ofclaim 1, wherein the final real-time video feed is outputted via an HDMIoutput.
 8. The method of claim 4, further wherein other parameters ofthe black box are configurable via the web page by pointing to the blackbox, wherein the other parameters include choices of processing means,locations of the remote server or the cloud, authentication information,security certificates, and various types of processing choices of theaudio and video signals, further wherein asaid processing choiceincludes mixing the video and audio signals such that the processedvideo and audio signals appear to originate from only a single source.9. The method of claim 1, further comprising a step of: filling areasvacated by the removed objects with neighboring pixels from a vicinityof the removed object.
 10. The method of claim 1, wherein the selectionsof objects to be removed are made by a user on a freeze frame of thescene from the first and the second cameras.
 11. The method of claim 1,wherein the defining of object contours is facilitated by a useroutlining a visualization of the object on the display using the inputdevice.
 12. A method for removing certain sensitive objects from a videofeed in real-time, the method comprising: receiving, at a multimediablack box having at least a touch screen, video feeds from at least twocameras, the cameras capturing substantially the same scene, havingsubstantially the same objects in a periphery, further wherein the atleast two cameras have associated input and output elements fortransmitting and receiving audio and video; displaying the video feedsfrom the at least two cameras on the touch screen; pausing the videofeeds momentarily such that contours of objects present in the scene areselectively definable via gestures made on the touch screen; combiningthe video feeds to produce a resultant video feed with the definedobjects removed or censored; and outputting the resultant video feed.13. The method of claim 12, wherein the resultant video feed isoutputted via an HDMI port.
 14. The method of claim 12, wherein theresultant video feed is sent securely to a remote server through cloudcomputing via network cables, bluetooth, wifi, 4G, and other wirelesstechnologies.
 15. The method of claim 12, further comprising a step of:removing the defined objects and filling in the void with neighboringpixels from the vicinity of each object.
 16. The method of claim 12,further comprising a step of: selectively reinstating a previouslyremoved object based on gestures performed on the touch screen.